Detecting Randomness with SLMs
Project on SFT-ing a SLM to detect randomness
I finally gave my first presentation on a research project this year! I have been dragging my feet on doing this for a while, but I am glad to have finally done it.
Every year, Microsoft holds a couple of conferences for internal employees only. I submitted a talk titled, 'Finding a needle in a stack of needles - Detecting randomness with SLMs.'
The synopsis was essentially: Can an SLM do a better job of detecting a 'random' string than traditional methods can? I used freq.py (https://github.com/MarkBaggett/freq) as a baseline. Freq.py is a tool created by Mark Baggett from SANS to measure entropy of a string based off of a dictionary wordlist. The model I chose was DistilBERT.
The resulting comparison reaffirmed my belief that an SLM can outperform our typical tooling:
| Model | F1 Score |
|---|---|
| SFT'd SLM | 92% |
| freq.py | 82% |
Some of the code can be found here https://github.com/cams-security/SLM-training
Employing SLMs in ways to replace or supplement traditional security controls is quite fun, I think that I will explore more projects in this vein.